The disclosure of the following priority application is herein incorporated by reference: Japanese Patent Application No. 9-285536, filed Oct. 17, 1997.
1. Field of Invention
The present invention relates to an image processing method and device used, for example, in a color imaging device, and also to a recording medium on which is recorded an image processing program, and especially relates to pixel interpolation processing technology.
2. Description of Related Art
Generally, when a color image is photographed by using, for example, a video camera, a digital still camera or the like, incident light is passed through a filter and specified wavelength components in the incident light are extracted. In this way, for example, the light of each basic color component of R (red), G (green) and B (blue) is obtained.
Various methods of color image photography are known, such as a single plate method, a two-plate method, a three-plate method and a multiple-plate method, depending on the number of installed imaging units, e.g., CCD image sensors, in a device. However, in lower cost devices, often a single imaging unit is used.
When light from an object is separated into the three color components, R, G and B and all the color components are simultaneously photgraphed with a single imaging unit, three kinds of filters corresponding respectively to red, green and blue are arranged in a plane. The incident light which passes through the filter is directed at the imaging unit.
The two-dimensional image which is transmitted to the imaging unit is formed of many pixels that are arranged in the vertical and horizontal directions. Each pixel of the two-dimensional image includes only red, green or blue color light since the light has passed through one of the three kinds of filters corresponding respectively to red, green and blue.
For example, when the type of filter is changed for every pixel, and the filters are cyclically aligned in the order of R, G, B, R, G, B . . . in a horizontal direction, the color of pixels that are aligned in the horizontal direction become R, G, B, R, G, B . . . .
Accordingly, the information of the photographed object can be obtained only once every three pixels for any color component for red, green or blue. In other words, the object cannot be photographed other than in units of three pixels.
For all the pixels of the two-dimensional image, a pixel interpolation is implemented in order to obtain the color components of the red, green and blue colors. Therefore, information corresponding to color components of a missing pixel is created based on, i.e., interpolated from, information corresponding to pixels in the vicinity of the missing pixel.
Generally, when this kind of pixel interpolation is performed, a value corresponding to the interpolated pixel is formed by simply averaging values corresponding to a plurality of pixels that exist around the position of the pixel which is to be interpolated. Moreover, there is also a case in which the interpolated pixel is formed from two pixel values that remain when the maximum and minimum value pixels are removed from 4 pixels that exist around the position of the pixel to be interpolated.
Technology for detecting an edge of an image with reference to surrounding pixels of different color components based on a color component of an interpolated pixel is well known.
Additionally, Japanese Laid-Open Patent Application 6-339145, U.S. Pat. No. 5,373,322, U.S. Pat. No. 5,053,861 and U.S. Pat. No. 5,040,064 disclose methods and devices in accordance with what is known in the art.
However, when an interpolated pixel is formed by simply averaging the values of the plurality of pixels existing around the interpolated pixel, a high frequency component of a spatial frequency of an image is easy to overlook. In other words, the edge component is easy to miss. Accordingly, resolution of a photographed image is decreased.
Moreover, when an image edge is detected with reference to surrounding pixels with color components different from the color component of the interpolated pixel, image shifting occurs in image areas where a luminance change and a color change do not match.
Further, in order to form an interpolated pixel with a comparatively high quality, processing including various complicated calculations is necessary. Accordingly, a resulting device becomes incredibly complicated and processing speeds become slow.
Accordingly, aspects of the present invention are directed at realizing formation of a high quality interpolated pixel by comparatively simple processing.
The present invention relates to image processing in which, for example, pixels of a first color component of an image are arranged at every other pixel in vertical and horizontal directions on a two-dimensional pixel array. Pixels of second color component and third color component are arranged uniformly throughout remaining positions in the two-dimensional pixel array. At each pixel position, an interpolated pixel is generated by interpolation when a pixel color component does not exist at that position. When a pixel at a target position where the given color component does not exist is interpolated, a value associated with one of primary reference pixels of the given color component located adjacent to the target position at the top and bottom or left and right of the target position is compared to values respectively associated with secondary reference pixels of the given color component existing in the vicinity of the primary reference pixel to detect an image edge component. When an image edge component is detected, an interpolated pixel is formed based on reference pixels of the given color component at the top and bottom or left and right of the target pixel. Interpolation is further performed by applying a weighting coefficient for reference pixels at the top and bottom of the target position and a weighting coefficient for reference pixels to the left and right of the target position. The top-bottom weighting coefficient is set to a different value than the left-right weighting coefficient.
According to one aspect of the present invention, image processing with improved image edge component detection may be performed by forming an interpolated pixel based on reference pixels of a given color component existing at positions adjacent to the top, bottom, left and right of a target position. A weighting coefficient of reference pixels on a detected image edge may be set to be larger than the other reference pixels. Additionally, a total sum of weighting coefficients of reference pixels may be normalized.
According to an aspect of the present invention, image processing with improved image edge component detection may be performed by forming an interpolated pixel based on two reference pixels of a given reference color component existing in positions on a detected image edge and adjacent to the top, bottom, left or right of a target position. The weighting coefficient of the reference pixels that do not exist on the edge may be set to zero.
According to an aspect of the present invention, image processing with improved image edge component detection may be performed by comparing each of two primary reference pixels of a primary color component existing at the top and bottom or left and right of a target position with four secondary reference pixels of the given color component existing in the vicinity of the primary reference pixels. An image edge component is detected when comparison of the primary color components and the given pixel color components fulfills a predetermined condition.
Additionally, the present invention provides an image processing apparatus that provides the above-described image processing with the improved image edge component detection.
Further, another aspect of the present invention provides a recording medium on which is recorded an image processing program, to be carried out by a computer, which performs the above-described image processing with the improved image edge component detection.
According to one aspect of the present invention, image data is assumed to be in a two-dimensional pixel arrangement such as, for example, a Bayer arrangement. A pixel of a first color component, for example, the G color component, is arranged in every other pixel in the vertical direction and in the horizontal direction, and pixels of a second color component, for example, the R color component, and pixels of a third color component, for example, the B color component, are equally arranged in the remaining positions.
Generally, the eyesight of a human is sensitive to the green wavelength, and is insensitive to the red and blue wavelengths. Accordingly, the value of the G color component is not greatly different from the luminance of the image which includes all the wavelengths of red, green and blue.
In a Bayer arrangement, two pixels of green color are arranged for every four pixels. Therefore, the proportion of G component pixels in the two-dimensional pixel arrangement is large. The resolution of the image is improved if this kind of pixel arrangement is adopted. Luminance component pixels that are pixels of a first color component which have a large pixel proportion in the two-dimensional pixel arrangement are interpolated. Performing suitable interpolation for pixels of the first color component effectively improves the resolution of the image.
In one aspect of this invention, when a pixel at a target position where the given color component does not exist is interpolated, a value associated with one of primary reference pixels of the given color component located adjacent to the target position at the top and bottom or left and right of the target position is compared to values respectively associated with secondary reference pixels of the given color component existing in the vicinity of the primary reference pixel to detect an image edge component. Specifically, adjacent positions at the top and bottom or left and right of the target position are checked for primary reference pixels. Additionally, the areas around the aforementioned primary reference pixels are checked for secondary reference pixels of the first color component. The existence of non-existence of an image edge component is identified.
When an image edge component is detected, an interpolated pixel is formed based on reference pixels of the first color component existing in adjacent positions at the top, bottom, left and right of the target position. Moreover, a weighting coefficient for the reference pixels at the top and bottom of the target position and a weighting coefficient for the reference pixels at the left and right of the target pixel are set at mutually different values.
In the vicinity of an edge position which extends in the vertical direction in the image, the change of brightness between adjacent pixels is large in the horizontal direction and small in the vertical direction. Moreover, in the vicinity of the position of an edge which extends in the horizontal direction in the image, the change of brightness between adjacent pixels is large in the vertical direction and small in the horizontal direction. Accordingly, when an edge in the vertical direction is detected, resolution in the horizontal direction is improved when influence of the reference pixels at the left and right of the target position is reduced and influence of the reference pixels at the top and bottom of the target position is increased. Similarly, when an edge in the horizontal direction is detected, resolution in the vertical direction is improved when influence of the reference pixels at the top and bottom of the target position is reduced and influence of the reference pixels at the left and right of the target position is increased.
Since existence or non-existence of the image edge component is detected by the simple comparison of the pixels, the ease of processing is improved and the time required to form the interpolated pixel is reduced. Moreover, since an interpolated pixel of a first color component is formed based on reference pixels of the first color component, even when shifting occurs between a position in which luminance changes and a position in which hue changes, an interpolated pixel is more indicative of the original image.
In another aspect of the invention, a weighting coefficient of the reference pixels on the detected edge is large and the weighting coefficient of the reference pixels which are adjacent to the detected edge is small. When an edge in the vertical direction is detected, the influence of the reference pixels at the left and right of the target position is reduced and the influence of the reference pixels at the top and bottom of the target position is increased. Accordingly, the resolution in the horizontal direction is improved. Moreover, when an edge in the horizontal direction is detected, the influence of the reference pixels at the top and bottom of the target position is reduced and the influence of the reference pixels at the left and right of the target position is increased. Accordingly, resolution in the vertical direction is improved. Therefore, the brightness of the interpolated pixel is suitably controlled by normalizing the weighting coefficients applied to the reference pixels to make the total sum of the weighting coefficients for the reference pixels become one.
In another aspect of the invention, when two reference pixels positioned on the detected edge significantly influence an interpolated pixel, the influence of the reference pixels positioned adjacent to the detected edge is nullified. Therefore, when an edge in the vertical direction is detected, there is no influence of the reference pixels at the left and right of the target position and the interpolated pixel is formed based on only the reference pixels at the top and bottom of the target position. Accordingly, the resolution in the horizontal direction is improved. When an edge in the horizontal direction is detected, there is no influence of the reference pixels at the top and bottom of the target pixel. Therefore, the interpolated image is formed according to only the reference pixels at the left and right of the target pixel. As a result, resolution in the vertical direction is improved.
In another aspect of the invention, each of two primary reference pixels of a first color component existing in adjacent positions at the top and bottom or the left and right of the target position are compared with our pixels of the first color component existing in the vicinity of the aforementioned reference pixels. When all these comparisons satisfy a predetermined condition, the image edge component is detected. For example, an edge in the vertical direction is considered as detected when a primary reference pixel positioned on the top side of the target position has a value associated with it that is larger than the values associated with the four secondary reference pixels that are in the vicinity of the one primary reference pixel. Moreover, an edge in the horizontal direction is considered as detected when one primary reference pixel positioned at the left side of the target position has a value associated with it that is larger than the values associated with the four secondary reference pixels that are in the vicinity of the one primary reference pixel.
According to another aspect of the invention, an interpolation controller forms an interpolated pixel in a target position where a pixel of a given color component does not exist. A value associated with one of primary reference pixels of the given color component located adjacent to the target position at the top and bottom or left and right of the target position is compared to values respectively associated with secondary reference pixels of the given color component existing in the vicinity of the primary reference pixel to detect an image edge component.
When an image edge component is detected, the interpolated pixel is formed based on the reference pixels of the first color component that exist in the adjacent positions at the top, bottom, left and right of the target position. Moreover, the weighting coefficient for the reference pixels at top and bottom of the target position and the weighting coefficient for the reference pixels at the left and right of the target pixel are fixed with mutually different values.
For example, it is considered that in the vicinity of an edge position which extends in the vertical direction in the image, the change of the brightness is large between the pixels that are adjacent to each other in the horizontal direction, and the change of brightness is small in the vertical direction. Moreover, in the vicinity of the position of an edge which extends in the horizontal direction in the image, the change of brightness between pixels that are adjacent to each other is large in the vertical direction, and the change of the brightness is small in the horizontal direction. Accordingly, when an edge in the vertical direction is detected, the influence of the reference pixels at left and right of the target pixel is reduced and the influence of the reference pixels at the top and bottom of the target pixel is increased and the resolution in the horizontal direction is improved.
Moreover, when an edge in the horizontal direction is detected, the influence of the reference pixels at the top and bottom of the target pixel is made to be small and the influence of the reference pixels at the left and right of the target pixel is made to be large and the resolution in the vertical direction is improved.
Since the existence or non-existence of an image edge component is detected by the simple comparison of the pixels, the processing is very easy, and the time necessary for forming the interpolated pixel is short. Moreover, an interpolated pixel of the first color component is formed based on reference pixels of the first color component, even when shifting occurs between the position at which the luminance changes and the position at which the hue changes, and an interpolated pixel which is faithful to the original image can be formed.
A recording medium which records the image processing program of the present invention is readable by a computer. The recording medium can be, for example, a CD-ROM, a RAM, a computer hard drive or a carrier wave in which the image processing program is transmitted as a computer data signal. Subsequently, the computer executes the image processing program. The interpolated pixel is formed as the computer follows this image processing program. When the interpolated pixel is formed at the target position at which a pixel of the first color component does not exist, the relative values between the primary reference pixels of the first color component existing in adjacent positions at the top and bottom or left and right of the target pixel and the secondary reference pixels of the first color component which exist in the vicinity of the aforementioned primary reference pixels are compared.
According to this comparison result, the existence or non-existence of an image edge component is identified. When the image edge component is detected, an interpolated pixel is formed based on the reference pixels of the first color component that exist in the positions that are adjacent at the top and bottom or left and right of the target pixel. In addition, the weighting coefficient for the reference pixels at the top and bottom of the target pixel and the weighting coefficient for the reference pixels at the left and right of the target pixel are set to mutually different values.
Since this image processing program can be widely executed without depending on specific hardware, the image processing program can be executed by reading-in with personal computers or work stations that are widely used.